An End-to-End Formula Recognition Method Integrated Attention Mechanism

Formula recognition is widely used in document intelligent processing, which can significantly shorten the time for mathematical formula input, but accuracy of traditional methods could be higher. In order to solve complexity an end-to-end encoder-decoder framework with attention mechanism proposed that converts formulas pictures into LaTeX sequences. The Vision Transformer (VIT) employed as encoder convert original input picture a set semantic vectors. Due two-dimensional nature formula, accurately capture characters’ relative position and spatial characteristics, positional embedding introduced ensure uniqueness character position. decoder adopts attention-based Transformer, vector translated target character. model joint codec training Cross-Entropy loss function, evaluated on im2latex-100k dataset CROHME 2014. experiment shows BLEU reaches 92.11, MED 0.90, Exact Match(EM) 0.62 dataset. This paper’s contribution introduce machine translation realize transformation from trajectory point sequence latex sequence, providing new idea based deep learning.